Eye movement analysis with co-clustering of hidden markov models (emhmm with co-clustering) and with switching hidden markov models (emshmm)

ABSTRACT

Provided are an Eye Movement analysis with Hidden Markov Model (EMHMM) with co-clustering, an Eye Movement analysis with Switching Hidden Markov Models (EMSHMM) to analyze eye movement data in cognitive tasks involving stimuli with different feature layouts and cognitive state changes, a switching hidden Markov model (SHMM) to capture a participant&#39;s cognitive state transitions during the task and an EMSHMM to assess preference decision-making tasks with two or more cognitive states. The EMSHMM provides quantitative measures of individual differences in cognitive behavior/style, making a significant impact on the use of eye tracking to study cognitive behavior across disciplines.

BACKGROUND OF THE INVENTION

Recent research has shown that people have idiosyncratic eye movementpatterns in visual tasks that are consistent across different stimuliand tasks (e.g., Andrews & Coppola, 1999; Castelhano & Henderson, 2008;Poynter, Barber, Inman, & Wiggins, 2013; Kanan, Bseiso, Ray, Hsiao, &Cottrell, 2015). These idiosyncratic eye movement patterns may reflectindividual differences in cognitive style or abilities. For example, itwas found that participants who demonstrated higher levels of curiositymade significantly more fixations in a scene viewing task than those whodemonstrated lower levels of curiosity. It was further found that whenviewing human faces, those who scored higher on extraversion andagreeableness personality traits looked at the eyes significantly moreoften than those who scored lower (Wu et al., 2014).

In recent years, there have been attempts of using machine-learningmethods to infer characteristics of the observer from eye movement data(e.g., Kanan et al., 2015). These studies typically use classifiers todiscover eye movement features important for distinguishing two or moreobservers, and do not tell us about overall eye movement patternsassociated with a particular observer. To better understand theassociation between eye movement patterns in visual tasks and individualdifferences in cognitive style or abilities, the inventors have recentlydeveloped a state-of-the-art eye movement data analysis method, EyeMovement analysis with Hidden Markov Model (EMHMM), which takesindividual differences in temporal and spatial dimensions of eyemovements into account (Chuk, Chan, Hsiao, 2014. EMHMM Matlab toolbox isavailable at http://visal.cs.cityu.edu.hk/research/emhmm/).

The EMHMM method is based on the assumption that during a visual task,the currently fixated region of interest (ROI) depends on the previouslyfixated ROI. Thus, eye movements in a visual task may be considered aMarkovian stochastic process, which can be better understood using ahidden Markov model (HMM), a type of time-series statistical model inmachine learning. More specifically, in this method, HMMs were used todirectly model eye movement data, and hidden states of the HMIscorresponded to ROIs of eye movements. The transition probabilitiesbetween hidden states (ROIs) represented the temporal pattern of eyemovements between ROIs. To account for individual differences, one HMMwas used to model one person's eye movement pattern in a visual task interms of both person-specific ROIs and transitions among the ROIs. Anindividual's HMM was estimated from the individual's eye movement datausing a variational Bayesian approach that can automatically determinethe optimal number of ROIs. In addition, individual HMIs could beclustered according to their similarities (Coviello, Chan, & Lanckriet,2014) to reveal common patterns. Differences among models could bequantitatively assessed using likelihood measures, that is, bycalculating the log-likelihood of data from one model being generated byanother model.

Thus, the EMHMM method was particularly suitable for examiningindividual differences in eye movement patterns and their associationswith other cognitive measures. Also, since HMM is a probabilistictime-series model, it worked well with a limited amount of data (e.g.,20 trials), which is in contrast to deep learning methods that requirelarge amounts of data to train effectively. Thus, the EMHMM approach wasespecially suitable for psychological research where data availabilityis limited or data collection is time consuming.

The EMHMM method was successfully applied to face recognition researchand discovered novel findings thus far not revealed by other methods.For example, two common eye movement patterns for face recognition werediscovered: a “holistic” pattern in which participants mainly looked atthe center of a face, and an “analytic” pattern that involved morefrequent eye movements between the two eyes and the face center (see,e.g., FIG. 1a ). Interestingly, analytic patterns were associated withbetter recognition performance, and this effect was consistentlyobserved across different culture and age groups (e.g., Chuk, Chan, &Hsiao, 2017; Chuk, Crookes, Hayward, Chan, & Hsiao, 2017; Chan, Chan,Lee, & Hsiao, 2018). In addition, significantly more participants (75%)used same eye movement patterns when viewing own- and other-race facesthan different patterns (Chuk, Crookes, et al., 2017). In contrast, onlyaround 60% of participants used the same eye movement patterns betweenface learning and recognition, and their recognition performance did notdiffer significantly from those using different patterns. This is incontrast to the scan path theory, which posits that eye movements duringlearning have to be recapitulated during recognition for the recognitionto be successful (Chuk et al., 2017).

It was further found that older adults adopted holistic patterns whereasyoung adults adopted analytic patterns (see, e.g., FIG. 1b ). Thisdifference was not readily observable from group eye fixation heat maps,demonstrating the power of the EMHMM method (FIG. 1c ; Chan et al.,2018). Among older adults, holistic patterns were associated with lowercognitive status as assessed using Montreal Cognitive Assessment(HK-MoCA; Yeung, Wong, Chan, Leung, & Yung, 2014); particularly inexecutive function and visual attention abilities (as assessed by Towerof London (TOL) and Trail Making Tests). Interestingly, this associationwas replicated when models of the discovered common patterns were usedto assess new participants' eye movement patterns when viewing new faceimages, suggesting the possibility of developing representative modelsfrom the population for cognitive impairment screening purposes.

Previous eye movement data analysis methods, including the use ofpredefined regions of interest (ROIs) or fixation heat map, did not taketemporal information of eye movements into account and the use ofpredefined ROIs might have involved experimenter biases. Further, inthese methods the analysis was typically performed on averaged or groupdata, and thus did not reflect individual differences in eye movementpattern. In addition, there was no quantitative measure of eye movementpattern similarity that took both spatial (eye fixation location) andtemporal information (sequence of fixations) into account.

Also, the previous EMHMM method was limited to visual tasks wherestimuli have the same feature layout (such as face recognition) and didnot involve cognitive state changes. Therefore, novel methods are neededfor quantitative measurements of eye movement patters that take spatialand temporal information into account and allow a more comprehensiveprediction of human behavior based on eye movement patterns.

BRIEF SUMMARY OF THE INVENTION

Provided are methods and systems for the analysis and modeling ofcognitive tasks involving cognitive state changes to predict subjectbehavior and/or cognitive processes. Specifically, provided is an EyeMovement analysis with Hidden Markov Model (EMHMM) with co-clusteringand an Eye Movement analysis with Switching Hidden Markov Model(EMSHMM). The models of the invention measure differences in eyemovement patterns by summarizing individuals' eye movements using hiddenMarkov models, allow discovery of common eye movement patterns, andprovide quantitative measures of eye movement pattern similarities thattake both temporal and spatial information of eye movements intoaccount. Advantageously, the EMSHMM of the instant invention allows dataanalysis and modeling for cognitive tasks involving cognitive statechanges and provides better predictions and analysis of subject behaviorand cognitive processes. Furthermore, the EMHMM with co-clusteringallows data analysis in tasks and cognitive behaviors that involvestimuli with different layouts, including, but not limited to, websiteviewing, information system usage, and visual search.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-C show the detection of two common eye movement patterns forface recognition. FIG. 1A shows a holistic eye movement pattern on theleft and an analytical eye movement pattern on the right. FIG. 1B showsa correlation of holistic and analytic eye movements with age. FIG. 1Cshows the fixation heat maps of old participants on the left and ofyoung participants on the right.

FIG. 2 shows an example of EMHMM with co-clustering of the invention.Circles indicate ROIs.

FIG. 3 shows histograms of symmetric KL (SKL) divergence between Group 1and Group 2 strategies on vehicle or animal images.

FIGS. 4A-C show examples of eye movement strategies. FIG. 4A showsexample stimuli. FIG. 4B shows eye movement strategies for the 2 groupson the stimuli where the 2 groups use strategies that have a largerdifference on the first 2 images and use strategies with a smallerdifference on the last 2 images as measured by symmetric KL divergence(SKL). FIG. 4C shows heat map plots of eye fixations on the first imagefor the 2 groups and the difference regions in orange for Group 1 andblue for Group 2.

FIGS. 5A-F show correlation analyses. FIG. 5A shows the correlationbetween EF scale and average number of fixations. FIG. 5B shows thecorrelation between EF scale and average saccade length. FIG. 5C showsthe correlation between EF scale and preference rating. FIG. 5D showsthe correlation between EF scale and scene recognition performance ind′. FIG. 5E shows the correlation between EF scale and Flanker CongruentAcc. FIG. 5F shows the correlation between EF scale and TOL planningtime before execution.

FIG. 6 shows an example SHMM model summarizing a participant's eyemovement pattern in the preference decision-making task. The blue arrowsindicate the transition probabilities between cognitive states(high-level states); the red arrows indicate the transitionprobabilities between the ROIs (low-level states).

FIG. 7 shows the gaze cascade plots during the last 2.5 seconds beforeresponse for two participant groups (Group A and B) and all participants(All). The red, blue, and green stars on the top indicate the timepoints at which the gaze cascade effect occurred; the black stars on thebottom indicate the time points during which the two groups hadsignificant differences in the proportion of time spent on the chosenitem.

FIGS. 8A-B show the differences in transitions between exploration andpreference-biased periods between two participant groups. FIG. 8A showsthe average probability of the two groups being in the preference-biasedperiod throughout a trial. FIG. 8B shows the probability distribution ofthe number of fixations in the exploration period (top) and the numberof fixations in the preference-biased period (bottom).

FIGS. 9A-B show the average inference accuracies of the two participantgroups. FIG. 9A shows the average inference accuracies using partialfixations in a trial, selected as a percentage of each trial's durationfrom the beginning. FIG. 9B shows the average inference accuracies ofthe two participant groups using different window lengths starting fromthe beginning of the trial.

FIGS. 10A-B show the average inference accuracies using differentfixation patterns. FIG. 10A shows the average inference accuracies oftwo groups of participants using fixations in the last 2 seconds beforethe participants' response. FIG. 10B shows the relationship betweenparticipants' eye movement patterns (measured in AB scale) and thecorresponding inference accuracy.

FIG. 11 shows the average inference accuracy using SHMMs and regularHMMs when using the last 2 seconds of the trials.

DETAILED DISCLOSURE OF THE INVENTION

Provided are methods and systems for the analysis and modeling ofcognitive tasks involving cognitive state changes to predict subjectbehavior and/or cognitive processes.

In some embodiments, an Eye Movement analysis with Hidden Markov Model(EMHMM) with co-clustering is provided where EMHMM is combined with thedata mining technique co-clustering to detect participant groups withconsistent eye movement patterns across stimuli for tasks involvingstimuli with different feature layouts. In specific embodiments, thismodel is applied to scene perception.

In some embodiments, an Eye Movement analysis with Switching HiddenMarkov Model (EMSHMM) is provided to detect participant groups withconsistent eye movement patters with cognitive state changes. The EMSHMMof the instant invention is particularly useful for tasks involvingdecision making where participants need to explore different options anddecide which option they prefer; a process that involves at least twocognitive states: exploration and decision making.

Also provided are methods for determining a subject's perception styleand cognitive abilities.

The EMHMM method of the invention takes individual differences intemporal and spatial dimensions of eye movements into account and usesHMMs to model eye movement data, where the hidden states of the modeldirectly correspond to regions of interest (ROIs) of eye movements.

In some embodiments, the instant methods summarize each person's eyemovement patters using an HMM in terms of both person-specific ROIs andtransitions among the ROIs and clusters the individual HMIs according totheir similarities. Differences among models are quantitatively assessedusing likelihood measures, which reflect similarities among individualpatterns.

In some embodiments, the EMHMM of the instant invention is applied toface recognition to assess cognitive status in executive and visualattention functioning and is used to assess cognitive decline ordeficits. For example, using the EMHMM method of the instant invention,a perception style can be assigned to an individual based on the eyemovements of the individual measured and evaluated using the EMHMM ofthe invention. Specific eye movement patterns measured with the EMHMM ofthe invention are associated with an analytic style of perception whileother eye movement patterns are associated with a holistic style ofperception. Furthermore, an analytic perception style is associated withimproved recognition performance while a holistic perception style isassociated with a cognitive decline in executive function and visualattention. Therefore, the methods of the instant invention can be usedto detect cognitive performance or the decline thereof in an individualbased on the individual's eye movement patterns. Importantly, themethods of the invention allow such cognitive performance assessmentwithout the need for determining additional physiological parameters.Therefore, the methods of the invention are suitable for applicationswhere detection of eye movement is the only data available such as indriver assessment and/or eye movement assessment via any computer orelectronic device screen.

In some embodiments, the EMHMM with co-clustering of the instantinvention is used to assess individuals that share similar eye movementpatterns across stimuli. For example, the EMHMM with co-clusteringmethod of the instant invention comprises the generation of individualHMMs including personalized ROIs and transition probabilities betweenROIs; the determination of common patterns through clustering; thequantification of similarities between patterns using datalog-likelihoods; and the use of the similarity measures to examine therelationship between eye movement patterns and cognitive measures.

In some embodiments, the EMHMM with co-clustering methods of theinvention assesses individuals' eye movement patterns across stimuliduring scene perception and defines an association of the eye movementpatterns with foreground object recognition performance and cognitiveabilities. In the EMHMM with co-clustering method of the invention theco-clustering formulation ensures that the participant groups areconsistent across all stimuli and the quantitative assessment usinglog-likelihood measures enables the determination of the relationshipsof eye movement patterns with a variety of cognitive measures. Using themethods of the instant invention, types of scene images that inducelarger individual differences in eye movement patterns can be determinedand object recognition performance and cognitive abilities ofindividuals be measured based on eye movement patterns during sceneviewing.

In some embodiments, EMHMMs of the instant invention use HMMs to modeleye movements, where the hidden states of the HMMs are directlyassociated with regions of interest (ROIs) of eye movements and thetemporal dynamics of eye movements are determined in terms of ROIs andtransitions among ROIs.

In some embodiments, individual HMMs are clustered to detectrepresentative strategies among subjects and co-clustering is used tocluster subjects into groups according to whether they use similar eyemovement strategies across stimuli. Differences between group strategiesare subsequently quantified using symmetric KL (SKL) divergence betweenthe group HMMs. Correlations between eye movement patterns andrecognition performance and/or cognitive abilities are determined byquantifying the similarities of a subject's pattern to the groupstrategies using a cluster score where log-likelihoods of a subject'sdata under the different group strategies are employed. For example,individuals' eye gazes across images are clustered into two groups usingthe data mining technique co-clustering resulting in two group HMMs foreach image. Histograms of SKL divergence between the two general patterngroups are generated, where SKL quantifies the difference between thetwo group patterns for each stimulus. Preferentially, images that inducelarger differences between individuals are used because largerindividual differences are more likely to provide adequate varianceamong individuals for identifying atypical patterns.

In preferred embodiments, cognitive ability tests are used together witheye movement pattern detection. For example, a Tower of London (TOL)test is used to test for measuring executive function, a Flanker Tasktest is used for measuring visual attention, and a Verbal & VisuospatialTwo-Back Task test is used for measuring working memory. Using thesemethods of the invention it has been determined that eye movementpattern in scene perception are particularly suited to measure visualattention and executive function. Therefore, in preferred embodiments,the EMHMM with co-clustering method of the invention is used to measurevisual attention and executive function in individuals, where a focusedeye movement pattern indicates superior visual attention and executivefunction compared to an explorative eye movement pattern. In furtherpreferred embodiments, the EMHMM with co-clustering method of theinvention is used to measure object recognition performance, where anexplorative eye movement patterns indicates superior object recognitionperformance compared to a focused eye movements pattern.

In some embodiments, hierarchical HMMs are provided with at least twolayers: a high-level HMM that acts like a switch that captures thetransitions between cognitive states and several low-level HMMs that areused to learn the eye movement pattern of a cognitive state.

Advantageously, the models of the invention measure differences in eyemovement patterns by summarizing individuals' eye movements using hiddenMarkov models, allow discovery of common eye movement patterns, andprovide quantitative measures of eye movement pattern similarities thattake both temporal and spatial information of eye movements intoaccount. For example, the EMSHMM of the instant invention allows dataanalysis and modeling for cognitive tasks involving cognitive statechanges and provides better predictions and analysis of subject behaviorand cognitive processes. Furthermore, the methods of the invention allowdata analysis in tasks and cognitive behaviors that involve stimuli withdifferent layouts, including, but not limited to, website viewing,information system usage, visual search, driving, and other complextasks.

In specific embodiments of the invention, the data may be received fromany type of eye tracking mechanism, whether optical, electrical,magnetic, or otherwise that calculate the eye gaze position of a humaneye. In preferred embodiments, the eye movements are recorded using anEyelink 1000 eye tracker, but any eye tracking mechanism may be used inperforming the instant invention without deviating from the spirit orscope of the invention. In preferred embodiments, eye tracking patternsare collected using video camera images.

In some embodiments, the data received from the eye tracking mechanismsare used in a Hidden Markov model. A Hidden Markov model is astatistical model used primarily to recover a data sequence that is notimmediately observable. The model derives probability values for theunobservable data sequence by interpreting other data that depend onthat sequence and are immediately observable.

In some embodiments, the Hidden Markov model of the instant inventionrepresents the visible output, e.g., the raw data received from theeye-tracking mechanism as a randomized function of the invisibleinternal state, e.g., the cognitive state of a subject.

The Hidden Markov model can be used to model visual attention or eyemovement changes corresponding to transitions in cognitive states duringa complicated cognitive task.

In specific embodiments of the instant invention, HMMs are used directlyto model eye movements, and hidden states of the HMMs are directlyassociated with ROIs of eye movements. This allows the determination oftemporal dynamics of eye movements in terms of ROIs and transitionsamong ROIs specific to an individual.

Advantageously, the novel methods of the instant invention detectmultiple cognitive states that occur during a task and the eye movementpattern associated with each cognitive state; thus allowing a betterunderstanding of individual differences in eye movement patterns inreal-life complicated cognitive tasks and predicting cognitive statesfrom eye movement patterns.

In some embodiments, the instant invention provides switching HMIs(SHMMs) that are hierarchical HMMs with two layers containing ahigh-level HMM and several low-level HMIs. Each low-level HMM can beused to learn the eye movement pattern of a cognitive state. Thehigh-level HMM acts like a ‘switch’ that captures the transitionsbetween cognitive states. It does so by learning the transitions betweenthe low-level HMMs.

The SHMMs of the instant invention are used to model transitions betweencognitive states and their associated eye movement patterns in acognitive task.

In some embodiments, the high-level states represent the cognitivestates, whereas the low-level states correspond to ROIs over thestimuli.

In some embodiments, eye movements are modeled in a preference decisionmaking task using face preference decision making in which participantsare presented with two face images in each trial and asked to judgewhich face they like more. According to embodiments of the invention, atleast two different eye movement patterns are observed one involving nofixation preference over either stimulus, which is related toexploration and information sampling, whereas the other involving ahigher percentage of fixations over the to-be-chosen stimulus, whichboth reflects and shapes a participant's preferences.

In specific embodiments, SHMM is used to capture the dynamics incognitive state and eye movement pattern. In some embodiments, one SHMMper individual is trained to summarize an individual's eye movementbehavior during a task. In some embodiments, two or more SHMMs aretrained using data from two different participant selections.

In preferred embodiments, individual SHMMs according to theirsimilarities are clustered to discover common eye movement patterns in atask. Different eye movement patterns are associated with differentdecision-making behavior and can be detected and measured with themethods and systems of the instant invention. Thus, the methods of theinstant invention can be used to infer, e.g., a decision making behaviorfrom eye movement patterns.

In some embodiments, the EMHMM with co-clustering is used to discovercommon patterns in participant groups and analyze tasks or cognitivebehavior involving stimuli with different layouts, including, but notlimited to, website viewing, information system usage, and visualsearch. Further tasks or cognitive behaviors that can be analyzed usingthe methods of the invention are reading, picture viewing, videoviewing, scene viewing, driving, navigation, and others.

Advantageously, the methods of the instant invention enable theassessment of more than one cognitive state in complex tasks and allowthe identification of said cognitive states based on the measured eyemovement patterns.

In preferred embodiments, specific eye movement patterns of a specificcognitive state are used to identify in a subject a certain cognitivestate.

In further embodiments, systems are provided to perform the methods ofthe instant invention. In some embodiments, the systems comprise acamera and a processor configured to detect an eye position within afacial image captured by the camera. In some embodiments, the processoris further configured to perform the steps and/or algorithms of theinstant invention. In preferred embodiments, the processor is configuredto read and process data from the camera according to a Matlab toolboxto analyze HMIs, EMHMMs, and EMSHMMs.

In some embodiments, an electronic device is provided that comprises acamera and a processor configured to detect a center position of an eyewithin a facial image captured via the camera in response to detectingan eye within the facial image; determine an eye gaze position based onthe center position and a pupil position; analyse the eye gaze positionin consecutively captured facial images; and measure an eye movementpatterns based on the eye gaze positions of the consecutively capturedfacial images.

In specific embodiments, the processor is further configured to generatean exploration transition matrix and Gaussian emission for each subject.In some embodiments, the processor is further configured to cluster orsummarize HMMs into single groups using a Variational HierarchicalExpectation Maximization algorithm, wherein the HMMs are clusteredaccording to their probability distributions. In other embodiment theprocessor is further configured to assimilate external user-entered datato associate certain eye movement patterns with certain external taskcriteria or cognitive state criteria.

In further embodiments, the processor is configured to determine thenumber of fixations of eye movements. Advantageously, from the eyemovement pattern and fixation probability distributions, the systems ofthe instant invention can calculate a probability of an individual beingin a certain cognitive state.

In some specific embodiments, the system of the invention can be used toinfer an individual's preference for a certain choice based on the eyemovement patterns.

In some embodiments, the system of the invention can be used todetermine an individual's cognitive style based on the eye movementpatterns of the individual. In specific embodiments, the processor ofthe system is configured to calculate a log likelihood of a eye movementpattern and assign a certain cognitive style to the measured eyemovement pattern if it matches the eye movement pattern of arepresentative group of individuals having said cognitive style.

Further provided are methods for inferring an individual's preferencechoice using eye movement patterns. In specific embodiments, theindividual's preference choice is inferred from the individual's eyemovement fixations measured in different time intervals from thebeginning of a trial to the making of a preference choice at the end ofthe trial. In specific embodiments, the individual's preference choiceis inferred from the individual's eye movement fixations measured duringthe last 25% of the time interval from the beginning of a trial. Inpreferred embodiments, the individual's preference choice is inferredfrom the individual's eye movement fixations measured during the last15% of the time interval from the beginning of a trial. In morepreferred embodiments, the individual's preference choice is inferredfrom the individual's eye movement fixations measured in the last 10% ofthe time interval from the beginning of a trial. In most preferredembodiments, the individual's preference choice is inferred from theindividual's eye movement fixations measured in the last 5% of the timeinterval from the beginning of a trial.

Further provided are methods for inferring an individual's preferencechoice using eye movement patterns measured at different time intervalsbefore a choice response. In specific embodiments, the individual'spreference choice is inferred from the individual's eye movementfixations measured in the last 10 seconds before the choice response. Inpreferred embodiments, the individual's preference choice is inferredfrom the individual's eye movement fixations measured in the last 8seconds before the choice response. In other preferred embodiments, theindividual's preference choice is inferred from the individual's eyemovement fixations measured in the last 6 seconds before the choiceresponse. In more preferred embodiments, the individual's preferencechoice is inferred from the individual's eye movement fixations measuredin the last 4 seconds before the choice response. In most preferredembodiments, the individual's preference choice is inferred from theindividual's eye movement fixations measured in the last 2 secondsbefore the choice response.

Advantageously, the instant methods enable the measurement of individualdifferences in eye movement patterns and cognitive styles during complexcognitive tasks. Furthermore, the instant methods allow the measurementof an individual's preferences significantly earlier than previouslyused methods. Specifically, the instant methods enable a measurement ofan individual's choice preference with only the first 75% of fixationsthat precede the choice.

Advantageously, the accuracy in inferring an individual's preferencechoices using the methods of the instant invention, particularly usingEMSHMM, is significantly higher than that using, e.g., EMHMM.

Furthermore, the EMHMM with co-clustering of the invention can be usedto infer an individual's cognitive style and/or cognitive abilities.

Importantly, the methods of the instant invention enable inferring anindividual's preference choice and cognitive style/abilities from eyegaze information alone using EMSHMM and/or EMHMM with co-clusteringwithout the need for further additional physiological information,making the instant methods and systems especially suited forapplications where eye movement measurements are the only data obtainedfrom an individual to determine the individual's cognitive style,inferring an individual's preference choice and/or measuring anindividual's cognitive abilities.

All patents, patent applications, provisional applications, andpublications referred to or cited herein are incorporated by referencein their entirety, including all figures and tables, to the extent theyare not inconsistent with the explicit teachings of this specification.

Following are examples that illustrate procedures for practicing theinvention. These examples should not be construed as limiting. Allpercentages are by weight and all solvent mixture proportions are byvolume unless otherwise noted.

Materials and Methods Example 1—Use of Eye Movement Analysis with HiddenMarkov Models (EMHMM) in Face Recognition

Using the EMHMM approach, the inventors previously detected holistic andanalytic eye movement patterns in face recognition through clustering ofeye movement patterns in 34 young and 34 older adults (Chan, Chan, Lee,and Hsiao (2018)). In the representative models, ellipses show ROIscorresponding to 2-D Gaussian emissions (FIG. 1A). Prior values show theprobabilities that a trial starts from the ellipse/ROI. Transitionprobabilities show the probabilities of observing a particulartransition to the next ROI from the current ROI. The small images on theright show the assignment of actual fixations to the ROIs and thecorresponding fixation heatmaps. Note that the clustering algorithm(Coviello, Chan, & Lanckriet, 2014) summarizes individual models in acluster into a representative HMM using a pre-specified number of ROIs,which may result in overlapping ROIs. Here the number of ROIs in therepresentative models was set to 3, the median number of ROIs in theindividual models. FIG. 1B shows the frequency of young and older adultsadopting holistic and analytic patterns: significantly more older adultsadopted holistic patterns and more young adults adopted analyticpatterns. FIG. 1C shows group fixation heat maps of older and youngadults.

Example 2—Use of EMHMM with Different Stimuli

Because the EMHMM method providing quantitative measures of individualdifferences in eye movement pattern is limited to tasks where stimulihave the same feature layout (e.g., faces), a novel method was developedthat combines EMHMM with the data mining technique co-clustering todetect participant groups with consistent eye movement patterns acrossstimuli for tasks involving stimuli with different feature layouts. Thisnovel method was applied to scene perception. Using the novel EMHMM withco-clustering method of the invention, explorative (switching betweenforeground and background information) and focused (mainly onforeground) eye movement strategies were detected. Explorative patternswere associated with better foreground object recognition performancewhereas those with focused patterns had better feature integration inthe flanker task and more efficient planning in the Tower of London(TOL) task. Advantageously, these novel methods can be used to employeye tracking as a window into individual differences in cognitiveabilities and for screening of cognitive deficits.

Scene perception involves complex and dynamic perceptual and cognitiveprocesses that can be influenced by both observers' goals and diversescene properties at multiple levels. Eye movements during sceneperception reflect the complexity of cognitive processes involved, andthus can potentially provide rich information about individualdifferences in perception styles and cognitive abilities. For example,it was observed that Westerners made more eye fixations on foregroundobjects whereas Asians looked at the backgrounds more often, and thisdifference was reflected in their foreground object recognitionperformance. This phenomenon has been linked to cultural differences inperception styles: Westerners are more likely to attribute the cause ofan event to isolated objects (analytic style), whereas Asians are morelikely to attribute the cause of an event to the context of the event(holistic style). However, these effects have not been foundconsistently and some studies found no difference between American andChinese participants either in foreground object recognition performanceor in number of fixations and durations to the foreground andbackground. Thus, it remains unclear whether eye movement patternsconsistently reflect cultural differences in perception styles andindividual differences in cognitive abilities.

Previous studies have attempted to use machine-learning methods to infercharacteristics of the observer from eye movement data. These studiestypically used classifiers to discover eye movement features importantfor distinguishing two or more observers. However, the classifiers onlylook for features important for separating the observers, and do notprovide any information about eye movement patterns associated with aparticular observer. The EMHMM approach, where HMM is a type oftime-series statistical model, has been developed by the inventors. Thisapproach takes individual differences in temporal and spatial dimensionsof eye movements into account. EMHMM assumes that during a visual task,the currently fixated region of interest (ROI) depends on the previouslyfixated ROI. Thus, eye movements may be considered a Markov stochasticprocess, which can be better understood using HMMs. Other studies haveused HMMs/probabilistic models for modeling eye movement/visualattention and cognitive behavior, where hidden states of the modelsrepresented cognitive states. In contrast, EMHMM of the instantinvention directly uses HMMs to model eye movement data, and hiddenstates of the models directly correspond to ROIs of eye movements. Toaccount for individual differences, each person's eye movement patternis summarized using an HMM in terms of both person-specific ROIs andtransitions among the ROIs, estimated from the individual's data using avariational Bayesian approach that can automatically determine thenumber of ROIs. Individual HMMs are clustered according to theirsimilarities to reveal common strategies. Differences among models arequantitatively assessed using likelihood measures, which reflectsimilarities among individual patterns. Thus, this method isparticularly suitable for examining individual differences in eyemovement patterns and their associations with other cognitive measures.Furthermore, because EMHMM is a Bayesian probabilistic model it workswell with limited amounts of data; in contrast to deep learning methodsthat require large amounts of data to train effectively. The EMHMM ofthe instant invention has been applied to face recognition research,discovering novel findings thus far not revealed by existing methods.For example, two common eye movement strategies were identified:analytic (more eyes-focused) and holistic nose-focused (FIG. 1A). Asiansand Caucasians did not differ significantly in the frequency of adoptingthe two strategies, suggesting little modulation from culture (Chuk etal., 2017). Analytic patterns were associated with better recognitionperformance, suggesting that retrieval of diagnostic information (i.e.,the eyes) is a better predictor for performance (Chuk, Chan, & Hsiao,2017). In contrast, older adults adopted holistic patterns compared toyoung adults, and their holistic patterns were associated with lowercognitive status particularly in executive and visual attentionfunctioning: the more holistic (nose-focused) the pattern, the lower thecognitive status (Chan, Chan, Lee, Hsiao, 2018). This correlation wasreplicated with new participants viewing new face images using the groupHMMs discovered from the old participants, suggesting the possibility ofdeveloping representative HMMs from the population for cognitivescreening purposes. Similarly, insomniacs' impaired ability for facialexpression judgments was associated with usage of a less eye-focusedpattern, suggesting that impaired visual attention control may accountfor compromised emotional face perception in insomniacs (Zhang et al.,2019). Together these results suggest the possibility of using eyetracking as a screening assessment for cognitive decline or deficits.

EMHMM has been limited to tasks involving stimuli with the same featurelayout so that discovered ROIs correspond to the same features acrossstimuli. For tasks where stimuli have different layouts, the inclusionof perceived images as features in the ROI representations has beenproposed but, while this method has improved discovery of ROIs, it doesnot work well when features among stimuli differ significantly, such asin scene perception.

Example 3—Novel EMHMM with Co-Clustering

The instant invention provides a new method that models eachparticipant's eye movements when viewing a particular stimulus with oneHMM, and uses the data mining technique co-clustering (see, e.g.,Govaert & Nadif, 2013) to detect subjects sharing similar eye movementpatterns across stimuli. The co-clustering formulation ensures that thesubject groups are consistent across all stimuli. The result is agrouping of subject and their representative HMIs for each stimulus(FIG. 2). The similarity between an individual's eye movement patternsacross stimuli and the representative HMMs are quantitatively assessedusing log-likelihood measures as in the existing approach. The resultsare used to examine the relationship between eye movement patterns andother cognitive measures. Using this method, individual differences ineye movement patterns during scene viewing were examined and it wasdetermined (1) what types of scene images (animals in nature vs.vehicles in a city) induce larger individual differences in eye movementpatterns, and (2) how eye movement patterns during scene viewing areassociated with subsequent foreground object recognition performance andcognitive abilities among subjects.

In a study, 61 Asian participants (35 females) aged 18-25 (M=20.77,SD=1.70) were recruited from the University of Hong Kong. Allparticipants had normal or corrected-to-normal vision. The materialsviewed consisted of 150 scene images with animals in a naturalenvironment and 150 scene images with vehicles in an urban environment.Images with different numbers of foreground objects, feature layouts,and locations of foreground objects were used to increase stimulusvariability to provide adequate opportunities to elicit individualdifferences in eye movement pattern.

The scene perception task consisted of a passive viewing phase and asurprise recognition phase (following Chua et al., 2005). During thepassive viewing phase, for each scene type (animals vs. vehicles),participants were presented with 60 images one at a time, each for 5seconds, and rated from 1 to 5 how much they liked the image. During thesurprise recognition phase, for each scene type, participants werepresented with 60 images with old foreground objects, with half of thempresented in the same old background and the other half in a newbackground, together with the same number of lure images with newobjects in a new background. Participants were presented with the imagesone at a time, and they judged whether they saw the foreground objectduring the passive viewing phase. The image stayed on the screen untilresponse. In both phases the animal and vehicle scene images werepresented in 2 separate blocks, with the block order counterbalancedacross participants. Participants' eye movements were recorded using anEyeLink 1000 eye tracker, with a chinrest to minimize head movements.Each trial started with a fixation cross at the screen center. Theexperimenter initiated the image presentation when a stable fixation wasobserved at the fixation cross. An image was then presented at thescreen center, spanning 35°×27° of visual angle at a viewing distance of0.6 m. Before each block, a 9-point calibration procedure was performed.Re-calibration took place whenever drift correction error exceeded 1° ofvisual angle.

In addition, participants performed 3 cognitive tasks to examine whethertheir eye movement patterns were associated with cognitive abilities:

(1) Tower of London (TOL) task (see, e.g., Phillips et al., 2001) fortesting executive function/planning abilities: In each trial,participants saw 3 beads randomly placed on 3 pegs as the startingposition, together with the target position. They were asked to move 1bead at a time to reach the target position as quickly as possible withthe minimum number of moves and to plan the moves in mind beforeexecution. In total there were 10 trials. The total number of extramoves, number of correct trials, total planning time before executingthe first move, and total execution time for the moves were measured.

(2) Flanker task (see, e.g., Ridderinkhof, Band, & Logan, 1999) fortesting selective attention: Participants judged the direction of anarrow flanked by 4 other arrows. In the congruent condition, theflanking arrows pointed at the same direction as the target arrow,whereas in the incongruent condition, the flanking arrows pointed at theopposite direction. In the neutral condition, the flankers werenon-directional symbols.

(3) Verbal and visuospatial 2-back task for assessing working memorycapacity (see, e.g., Lau et al., 2010): In each trial, participantsjudged whether the presented symbol/symbol location was the same as theone presented 2 trials back in the verbal/visuospatial taskrespectively.

Participants' eye movements during the passive viewing phase wereanalyzed using EMHMM with co-clustering (adapted fromhttp://viral.cs.cityu.edu.hk/research/emhmm). Each participant's eyemovements for viewing each stimulus were summarized using an HMM.Individual HMMs for viewing each stimulus were clustered to discover 2representative strategies among the participants. Co-clustering was thenused to cluster participants into 2 groups according to whether theyused similar eye movement strategies across the stimuli (FIG. 2). Toexamine whether natural or man-made images could induce largerindividual differences in eye movement pattern, the difference betweengroup strategies for each stimulus were quantified via the symmetric KL(SKL) divergence between the two group HMMs. The SKL is given by(KL₁₂+KL₂₁)/2, where KL₁₂ is the KL divergence between the Group 1 HMMand the Group 2 HMM using Group 1 data (Chuk, Chan, & Hsiao, 2014), andvice-versa for KL₂₁ (KL is not symmetric). SKL measures were comparedfor natural and man-made images and the characteristics of the imagesthat typically led to larger SKL were determined.

It was also examined whether the 2 groups of participants differed inperformance in foreground object recognition and the cognitive tasks. Toexamine the correlations between eye movement pattern and recognitionperformance/cognitive abilities, the similarity of a participant'spattern to the group strategies were examined using a cluster score,CS=(|L₁|−|L₂|)/(|L₁|+|L₂|), where L₁ and L₂ are the log-likelihoods of aparticipant's data being generated under Group 1 and Group 2 strategy,respectively (Chan et al., 2018). Larger positive values of CS indicatehigher similarity to Group 1, and smaller negative values indicatesimilarity to Group 2.

Example 4—Eye Movement Strategies in Scene Perception Determined UsingEMHMM with Co-Clustering

The data obtained as described in Example 3 from 61 participants wereclustered into 2 groups using EMHMM with co-clustering on the 120 imagestimuli. Group 1 contained 37 participants and Group 2 contained 24participants. The co-clustering model estimates 2 HMMs for each imagestimulus, corresponding to the strategies of Groups 1 and 2. For eachimage, the difference between the strategies was measured in SKL. Asshown in FIG. 3, animal images induced larger differences in eyemovement strategies between the 2 groups than vehicle images,t(118)=−5.626; p<0.0001. FIG. 4A presents 4 example images and theircorresponding HMMs for the 2 groups. FIGS. 4b 1 and 4 b 2 show twoexamples of eye movement strategies with large SKL difference betweenthe 2 groups. In particular, Group 2 focused more on the foregroundobject, while Group 1 explored the image, looking at both the foregroundobject and the background. In addition, while looking at an animal,Group 2 focused more on the eyes, whereas Group 1 looked at the nose(FIG. 4c ). FIGS. 4b 3 and 4 b 4 show examples where the 2 groups hadsimilar eye movement strategies. In general, larger differences in eyemovements occurred on images where the foreground object (animal or car)was salient as compared with the background, e.g., an animal amongtrees, or a car on a road. This indicates that animals are more salientthan vehicles, and that animal images generally induce largerdifferences in eye movement strategies than vehicle images. In contrast,images with cluttered backgrounds and non-interesting foreground objectstypically induced similar eye movement strategies. Group 1 and 2strategies are referred to as the explorative and focused strategy,respectively, and the cluster score between the two strategies as theExplorative-Focused (EF) scale. Indeed, participants using theexplorative strategy had a larger average number of fixations,t(59)=3.793, p<0.001, and longer saccade lengths, t(59)=4.881, p<0.001,than those using the focused strategy (Table 1 below), and the moresimilar their eye movement pattern to the explorative strategy (in EFscale), the larger the average number of fixations, r(60)=0.477,p<0.001, and saccade length, r(60)=0.544, p<0.001 (FIGS. 5a-e ). AsEMHMM does not use sequence length information, the difference inaverage number of fixations emerged naturally as the result of theclustering. These results were consistent with the interpretation thatGroup 1 was more explorative than Group 2.

TABLE 1 Comparison of eye movement statistics and cognitive taskperformance between the explorative and focused strategies (FOR:Foreground Object Recognition). Group 1 Group 2 Explorative Focused TaskMeasure mean std mean std p t Avg Number of 14.06 1.659 12.43 1.6020.000 3.793 Fixations Avg Fixation 312.4 43.23 335.9 66.77 0.100 −1.672Duration Avg Saccade Length 91.92 13.87 74.34 13.56 0.000 4.881 ImagePreference 2.545 0.485 2.890 0.496 0.009 −2.689 Rating FOR Reaction Time1160 303.4 1216 412.4 0.545 −0.609 (RT) FOR d’ Original 1.695 0.5561.050 0.922 0.001 3.410 Background FOR d’ New 0.802 0.302 0.531 0.4060.004 2.986 Background FOR d’ Overall 1.203 0.356 0.766 0.638 0.0013.434 N-back Verbal d’ 2.867 0.550 2.640 0.725 0.169 1.391 N-back VerbalRT 731.8 169.2 706.2 127.6 0.529 0.633 N-back Spatial d’ 2.244 0.7642.201 0.594 0.819 0.230 N-back Spatial RT 780.2 220.3 742.9 146.2 0.4680.730 Flanker Incongruent 89.26 14.79 90.21 19.56 0.830 −0.216 AccFlanker Congruent 98.85 2.091 99.79 0.706 0.038 −2.121 Acc FlankerNeutral Acc 97.97 3.272 97.81 3.240 0.852 0.188 Flanker Overall Acc95.36 5.331 95.4 6.677 0.710 −0.374 Flanker Incongruent 414.8 47.03415.3 99.80 0.980 −0.025 RT Flanker Congruent 362.5 35.08 373.1 36.370.261 −1.135 RT Flanker Neutral RT 366.0 37.88 373.6 36.39 0.440 −0.778Flanker Overall RT 381.1 38.23 387.4 48.54 0.579 −0.558 TOL Total #Moves 24.97 17.00 26.29 18.79 0.779 −0.282 TOL # Correct 5.194 2.5504.875 2.983 0.659 0.444 Trials TOL Total 177.7 69.23 181.8 77.44 0.829−0.216 Executaion Time (s) TOL Preplanning 137.0 71.33 99.21 36.27 0.0202.395 Time (s)

Example 5—Association Between Eye Movement Pattern and RecognitionPerformance/Cognitive Abilities Using EMHMM with Co-Clustering

Participants using the explorative strategy also had significantly lowerimage preference ratings, t(59)=−2.689, p=0.009; better foregroundobject recognition in d′, t(59)=3.434, p=0.001; lower accuracy incongruent trials of the flanker task, t(59)=−2.121, p=0.038; and longertotal planning time before execution in the TOL task, t(58)=2.395,p=0.02 (Table 1; one participant's TOL result was missing due totechnical problem). Note that their advantage in foreground objectrecognition performance was significantly larger when the foregroundobject was presented with the original background than with a newbackground, F(1, 59)=9.126, p=0.004, although the advantage wassignificant in both cases (Table 1). Consistent with these findings,correlation analysis showed that participants' eye movement patternsimilarity to the explorative strategy (EF scale) was correlatednegatively with preference ratings, r(60)=0.301, p=0.018; positivelywith scene recognition performance in d′, r(60)=0.381, p=0.002; andmarginally positively correlated with TOL planning time before executingmoves, r(59)=0.223, p=0.08 (FIGS. 5a-e ). Taken together these findingsindicate that the explorative strategy is associated with lower imagepreference rating, better scene recognition performance, decreasedfacilitation from consistent surrounding cues in the flanker task, andless efficient planning in the TOL task.

Overall, for images containing animal faces, participants adopting thefocused strategy (analytic style) looked more to the eyes of the animalfaces, suggesting engagement of local attention. In contrast, thoseadopting the explorative strategy (holistic style) looked more to theanimal face center, suggesting engagement of global processing. Theseresults are generally consistent with Caucasians' analytic style that isassociated with focusing on the foreground object in scene perceptionand the eyes in face recognition, whereas Asian's holistic style isassociated with looking more often at the background and the facecenter. However, using EMHMM, the inventors showed that Asians andCaucasians did not differ in the frequency of adopting the eyes-focusedor nose-focused strategy, suggesting little modulation from culture oneye movement pattern when individual difference was taken into account(Chuk et al. 2017).

It was observed that participants adopting the explorative strategy hadbetter foreground object recognition performance than those using thefocused strategy, regardless of whether the foreground object appearedin a new or old background; this advantage was larger when theforeground object appeared in the old than a new background. The findingsuggests that a more explorative eye movement strategy during sceneperception may be beneficial for remembering the foreground object dueto more retrieval cues available through exploration. Consistent withthis suggestion, it has been shown that associative processing isinherent in scene perception suggesting that an explorative strategy mayfacilitate associative processing and consequently enhance scene memory.This finding is in contrast to the face recognition literature, wherethe eye-focused strategy, which is associated with engagement of localattention and the focused strategy reported here, was reported to leadto better recognition performance due to better retrieval of diagnosticfeatures, the eyes.

In contrast, using the methods of the invention, it was shown that thoseadopting the focused strategy performed better in the congruent trialsof the flanker task and had shorter preplanning time in the TOL taskthan those using the explorative strategy. The advantage observed in thecongruent but not incongruent or neutral trials in the flanker tasksuggested that these participants might have better feature integrationabilities. The shorter preplanning time in the TOL task withoutdifferences in number of moves, correct trials, or execution timesuggested that they had more efficient planning abilities. Thus, becauseparticipants adopting the focused strategy preferred to look at the eyesof the animals, the focused strategy may be related to a moreeyes-focused strategy in face recognition, which was associated withbetter face recognition performance, visual attention (the trail makingtask), and executive function (the TOL task; Chan et al., 2018).

Using the methods of the invention it was also observed that images witha salient foreground object relative to the background tended to inducelarge individual differences in eye movement patterns, and that animalimages induced larger individual differences than vehicle images (FIG.3). This phenomenon may be due to humans' category specific attention toanimals that made them more salient than vehicles or other object types,providing better opportunities to induce the difference between theexplorative and focused strategies. This finding has importantimplications for the possibility of using eye tracking to provide screentools for cognitive disorders because images that induce largerindividual differences will be more likely to provide adequate varianceamong individuals for identifying atypically patterns.

In conclusion, it was show that the novel EMHMM with co-clusteringmethod of the invention can effectively summarize and quantitativelyassess individual differences in eye movement strategies in tasksinvolving stimuli with different feature layouts, and in turn lead tonew discoveries not yet found by existing methods. By applying the novelmethod to scene perception, the explorative and focused strategies amongAsians were detected. Whereas the explorative strategy was associatedwith better foreground object recognition performance, the focusedstrategy was associated with better feature integration and planningabilities. Also, images with a salient foreground object relative to thebackground induced larger individual differences in eye movementpatterns. These results have important clinical and educationalimplications for the use of eye tracking in cognitive deficit detectionand cognitive performance monitoring. Advantageously, the novel EMHMMwith co-clustering method of the invention can be applied to a largevariety of visual tasks and be instrumental in using eye movementpatterns to understand cognitive behavior.

Example 6—Eye Movement Analysis with Switching Hidden Markov Models(EMSHMM) in the Detection of Different Cognitive States Using a FacePreference Decision-Making Task

In previous probabilistic approaches to model visual attention in acomplicated cognitive task, the hidden states of the models representedcognitive states, thus capturing the temporal dynamics of cognitivestate transitions but not the dynamics of eye movements. In order tomeasure multiple cognitive states that occur during a task and the eyemovement patterns associated with each cognitive state an EMHMM withhierarchical HMMs is used. For example, the hierarchical HMM with twolayers contains a high-level HMM and several low-level HMIs. Eachlow-level HMM acts like a “switch” that captures the transitions betweencognitive states. It does so by learning the transitions between thelow-level HMMs. This method of the instant invention is called aSwitching HMM (SHMM). The SHMM of the invention is used to capture thedynamics in cognitive state and eye movement pattern following anexisting EMHMM approach and training one SHMM per participant tosummarize the participant's eye movement behavior during a task. Theindividual SHMMs are then clustered according to their similarities todetect common eye movement patterns in the task and examine thedifferent eye movement patterns associated with differentdecision-making behaviors.

Applying the novel EMSHMM of the instant invention, eye movement datawere collected from a face preference decision-making task and twoparticipant groups were compared. The preference decision-making taskwas a two-alternative-forced-choice task, which contained two parts. Inpart 1, participants were presented with 120 (female and male) computergenerated, bald faces. They were asked to rate, from 1 to 7, howattractive the faces were. These ratings were used for pairing stimulifor part 2 and the eye movements were not analyzed. After part 1 wasfinished, the faces of the same gender that received similar ratingswere paired to form 60 pairs as the stimuli in part 2. In part 2, ineach trial, each pair of faces was shown on the screen with one on theleft and one on the right. Participants were required to indicate whichface they preferred. There was no time limit. Participants could movetheir eyes freely to compare the two images. They were told to press abutton to indicate which face (left or right) they preferred once theyhad made their decision. Eye movements were recorded using an Eyelink1000 eye tracker. In data acquisition, fixation location information wasextracted using Eyelink Data Viewer. Saccade motion threshold was 0.15degree of visual angle; saccade acceleration threshold was 8000degree/square second; saccade velocity threshold was 30 degree/second.These are the EyeLink defaults for cognitive research.

Example 7—Switching Hidden Markov Model

A standard hidden Markov model (HMM) contains a vector of prior valuesof the hidden states, a transition matrix of the hidden states, and aGaussian emission for each hidden state. The prior values indicate theprobabilities of the time-series data starting from the correspondinghidden states. The transition matrix indicates the transitionprobabilities between any two hidden states. The Gaussian emissionsindicate the probabilistic associations between the observed time-seriesdata and the hidden states. In the previous EMHMM approach (Chuk et al.,2014), the hidden states corresponded to the regions of interest (ROIs),the emissions were the eye fixation locations, and the emissions in anROI were represented as a 2-D Gaussian distribution (see FIG. 1a ).

In contrast to a standard HMM, a switching HMM (SHMM) contains twolevels of hidden state sequences; the low-level hidden state sequencemodels the temporal pattern of the time-series data following a standardHMM, while the high-level hidden state sequences indicate thetransitions between the HMM parameters used by the low-level hiddenstate sequence. Formally, z_(n,t)∈{1, . . . , K} are the low-levelhidden states, and s_(n,t)∈{1, . . . , S} the high-level hidden states,and x_(n,t) the observations, where n indexes the sequences and tindexes time. Both the high-level state sequence and the low-level statesequence are 1^(st)-order Markov chains. The prior probability andtransitions of the low-level hidden state depends on the currenthigh-level state,

p(s _(n))=p(s _(n,1))Π_(t=2) ^(τ) ^(n) p(s _(n,t) |s _(n,t-1))  (1)

p(z _(n) |s _(n))=p(z _(n,1) |s _(n,1))Π_(t=2) ^(τ) ^(n) p(z _(n,t) |z_(n,t-1) ,s _(n,t)),  (2)

where τ_(n) is the length of the n-th sequence. The high-level statesequence is parametrized by the prior vector ρ and transition matrix B,

p(s _(n,1) =j)=ρ_(j) , p(s _(n,t) =j|s _(n,t-1) =j)=b _(i,j′).  (3)

Given that the high-level state is s_(n,t)=j, the low-level statesequence is parametrized by the prior vector π^((j)) and transitionmatrix A^((j)),

p(z _(n,1) =k|s _(n,1) =j)=π_(k) ^((j)) , p(z _(n,t) =k|z _(n,t-1) =k,s_(n,t) =j)=a _(k,k′) ^((j)).  (4)

The emission densities are Gaussians and depend only on the low-levelhidden state (i.e., they are shared among high-level states),

p(x ^(n,t) |z _(n,t) =k)=N(x _(n,t)|μ_(k),Λ_(k) ⁻¹),  (5)

where μ_(k), Λ_(k) ⁻¹ are the mean vector and covariance matrix of theGaussian. It was assumed that the number of low-level states for eachhigh-level state is the same. The joint probability model for the SHMMis

p(X,Z,S)=Π_(n=1) ^(D)[p(s _(n,1))p(z _(n,1) |s _(n,1))p(x _(n,1) |z_(n,1))Π_(t=2) ^(τ) ^(n) p(s _(n,t) |s _(n,t-1))p(z _(n,t) |z _(n,t- 1),s _(n,t))p(x _(n,t) |z _(n,t))].   (6)

In practice, the SHMM can be turned into a standard HMM by combining thehigh-level and the low-level hidden state variables into a single hiddenstate variable, whose values are the Cartesian product of the low- andhigh-level state values. Here, it was assumed that the low-level stateswere shared among the high-level states (S), and thus the number oflow-level states (K) was the same for each high-level state.Consequently, the equivalent standard HMM had S*K augmented hiddenstates {tilde over (z)}_(n,t). The augmented states took a value pair(j,k), where j indicates the high-level state and k indicates thelow-level state. The transition probabilities and the prior valuestherefore were defined as,

p({tilde over (z)} _(n,t)=(j′,k′)|{tilde over (z)} _(n,t-1)=(j,k))=ã_((j,k),(j′,k′)) =b _(j,j′) a _(k,k′) ^((j′)),  (7)

p({tilde over (z)} _(n,1)=(j,k))={tilde over (π)}_((j,k))=ρ_(j)π_(k)^((j)).  (8)

Thus the relationship between the augmented hidden states and the twoseparate levels of hidden state sequences were defined as

p({tilde over (z)} _(n))=p(z _(n) ,s _(n))=p(z _(n) |s _(n))p(s_(n))=p(z _(n,1) |s _(n,1))p(s _(n,1))Π_(t=2) ^(τ) ^(n) p(z _(n,t) |z_(n,t-1) ,s _(n,t))p(s _(n,t) |s _(n,t-1))=p(s _(n,1) ,z _(n,1))Π_(t=2)^(τ) ^(n) p(s _(n,t) ,z _(n,t) |s _(n,t-1) ,z _(n,t-1))  (9)

The transition matrix and the prior vector had block structures,

$\begin{matrix}{{\overset{\sim}{A} = \begin{bmatrix}{b_{1,1}A^{(1)}b_{1,1}A^{(2)}} \\{b_{2,1}A^{(1)}b_{2,2}A^{(2)}}\end{bmatrix}},} & (10)\end{matrix}$ $\begin{matrix}{\overset{\sim}{} = {\begin{bmatrix}{\rho_{1}\pi^{(1)}} \\{\rho_{2}\pi^{(2)}}\end{bmatrix}.}} & (11)\end{matrix}$

In the implementation, the high-level hidden states represent thecognitive states, whereas the low-level hidden states correspond to ROIsover the stimuli. The high-level state sequence has its own transitionmatrix, which governs the transitions between cognitive states. Thelow-level states (ROIs) are shared among the high-level states. Takingthe preference decision-making task as an example, it was assumed thatparticipants have two cognitive states: an exploration period thatinvolves information sampling without preference to a specific stimulus,and a preference-biased period where preference is formed and eyemovement behavior can be influenced by the preference. A simplifieddecision process was assumed where a participant started in theexploration period, and that once the participant had sampled enoughinformation in the exploration period, they transitioned to thepreference-biased period and could not transition back to theexploration period. That is, once the preference-biased period wasentered it could not be exited until the response decision. To examine agaze cascade effect, that is a gaze bias towards the later-selectedobject (see, e.g., Shimojo et al. 2003), it was assumed that thelow-level HMM had two ROIs, each corresponded to a stimulus for choice.FIG. 6 illustrates an example model summarizing a participant's eyemovement pattern. As shown in the figure, the high-level HMM consistedof two cognitive states as its hidden states: exploration andpreference-biased periods. The blue arrows indicate the transitionsbetween the two states, and the numbers indicate transitionprobabilities. Eye movements within each state were modeled with alow-level HMI, whose hidden states represent ROIs of eye movements. Thered arrows represent transitions between ROIs. The two cognitive stateshave the same ROIs but different transition probabilities.

Example 8—Training Individual SHMMs

An Expectation-Maximization (EM) algorithm was performed to estimate theSHMM parameters. In the Expectation step (E-step), the responsibilitieswere calculated using the standard forward-backward algorithm with theblock transition matrix, initial state vector, and emission densities.In the Maximization step (M-step), the prior and pairwiseresponsibilities were summed over the high-level and the low-levelstates, respectively, to yield the parameter updates for both thehigh-level states and the low-level states.

For example, the prior responsibilities were summed over the low-levelhidden states for each of the high-level state to yield the parameterupdates for the low-level state sequence, and then they were summed overthe high-level states to yield the parameter updates for the high-levelstate sequence. Similarly, the pairwise responsibilities were summedover the low-level hidden states for each high-level state to yield theupdates for each low-level transition, and then were summed over thehigh-level hidden states to yield the updates for the switching(transition) matrix of the high-level state sequence.

For each participant, two SHMMs were trained; one using the data fromthe left-selected trials and one using the data from the right-selectedtrials. The two SHMMs were combined into one for each individual asfollows. Preliminary analysis indicated that the exploration periods ofthe left-selected and right-selected trials were similar. In otherwords, the eye movements during exploration periods were consistentregardless of which side was selected. Thus, the transition matrices forthe exploration period of the left-selected and right-selected modelswere directly averaged together. For the preference-biased period, thetwo preferred-side parameters and the two non-preferred-side parameterswere averaged, essentially normalizing the right-selected trials intoleft-selected trials. In order to focus the analyses on the transitionbetween the stimuli during preference decision making, only two Gaussianemissions were used per model, one on each side, for the low-levelstates (FIG. 6). For SHMM estimation, one Gaussian centered over eachstimulus was initialized, with covariance that covered the stimulus.Thus, it could be considered here that the low-level states of the modelwere pre-specified (and thus not hidden), since it could be determinedwith good confidence which stimulus was viewed. The advantage of usingGaussian rather than discrete emissions is that it can be easilyextended to analyses that explore more ROIs (i.e., more hiddenlower-level states) on each stimulus. Two high-level hidden states wereused to reflect that the participants would transition from theexploration period to the preference-biased period during a trial.

For SHMM estimation, the transition matrices of the high-level statewere initialized as [0.95, 0.05; 0.0, 1.0] and high-level prior as [1.0,0.0], which encodes the assumed behavior of starting in the explorationperiod and staying there (0.95), and then eventually transitioning tothe preference-biased period (0.05) and not back (0.00). Duringtraining, this initialization causes the probability of transitioningfrom the preference-biased to the exploration period to stay at 0 (sinceall potential sequences that transition from the preference-biased tothe exploration period are given probability zero). The transitionmatrices of the low-level states were initialized as uniformdistributions. After the initialization, the Gaussian ROIs andtransition probabilities were updated in the EM algorithm.

Example 9—Clustering to Discover Common Patterns

To examine the general eye movement pattern adopted by all individualsduring the exploration period, an HMI was created using the explorationtransition matrix and Gaussian emissions for each individual, and theseHMIs were clustered or summarized into one group using the VariationalHierarchical Expectation Maximization (VHEM) algorithm (see, e.g.,Coviello et al., 2014). The VHEM algorithm clusters HMIs into apredefined number of groups according to their probability distributionsand characterizes each group using a representative HMM. A similarprocedure was performed to obtain the general eye movement pattern forthe preference-biased period.

To examine individual differences in decision making behavior, theparticipants' high-level transition matrices were clustered into 2groups using the k-means clustering algorithm (see, e.g., MacQueen,1967). The aim was to discover differences in the high-level cognitivebehavior involving the explorative and preference-biased periods. Foreach group, representative exploration-period andpreference-biased-period HMIs were computed by running the VHEMalgorithm on the exploration-period and preference-biased-period HMMs ofthat group respectively¹. It was then examined how participants in thetwo groups differed in decision making behavior, including theirdifferences in the gaze cascade plot/effect, transition between theexploration and preference-biased periods, distribution of number offixations in exploration/preference-biased periods, and accuracy ofinferring preference choices from eye movement data. ¹ Since thedifferences in the transitions between the two sides were of interest,all HMMs in this example were forced to use the same set of ROIs thatcovered each face.

Example 10—Transition Between Exploration and Preference-Biased Periods

After participants' high-level transition matrices were clustered intotwo groups, it was further investigated how the participants in thesetwo groups differed in decision-making behavior. More specifically, theprobability that the participant was in the preference-biased period wasinvestigated from the beginning to the end of a trial. For each trial,the posterior probabilities of all possible high-level state sequenceswere computed given the observed eye gaze data. Then a sequence ofprobabilities of being in the preference-biased period was computed bycomputing the expectation over the high-level state sequences, i.e., bycomputing the weighted average over all high-level state sequences wherethe weights are the posterior probabilities. This was performed on alltrials in all participants. Since the duration differed in each trial,to examine the proportion of time a participant spent in the explorationand preference-biased periods relative to the whole trial, the differentdurations were normalized across the trials by dividing each trial intothe same number of segments (21 in the instant experiments). Then, foreach segment in a trial, the probability of time that the participantwas in the preference-biased period was calculated. For eachparticipant, the mean probability was calculated across all trials foreach segment. Finally, the means across participants in the same groupwere averaged and the mean probability at each trial segment was plottedfor the two participant groups separately. The plot represented thepercentage of time at each trial segment the participants were in thepreference-biased period. This examination allowed the detection of thedifference in temporal dynamics of cognitive state changes in a trialbetween groups.

The number of fixations in the exploration and preference-biased periodbetween the two groups were also examined. To this end, for each trialand each participant, the posterior probabilities of all high-levelstate sequences were computed, and then the numbers of fixations in theexploration and preference-biased periods counted. The aggregatedprobabilities over all trials and all high-level sequences were thenused to form a probability distribution of number of fixations in eachhigh-level state for a participant. The participants' probabilitydistributions were then averaged together in each group.

Example 11—Inferences of Individual Preferences

It was examined whether SHMMs could be used to infer an individual'spreference choice in a trial. For each participant, the trials in thepreference decision making task were split into two sets: one for thetrials in which the left-side image was chosen to be preferred, and theother for the trials in which the right-side image was chosen. Similarto the preference decision making task, face images used in a trial werematched in gender and attractiveness ratings. Thus, the two sides wereexpected to be chosen equally often. For each set, all but one trial wasused to train a left-selected and a right-selected SHMM and the held-outtrial was used for testing. For testing, an aggregated SHMM was createdfrom the two trained SHMMs, which could be used to infer theparticipant's choice. In particular, the aggregated SHMM contained 3high-level states: exploration, left-selected preference-biased, andright-selected preference-biased periods. In the high-level transitionmatrix of this aggregated SHMM, the transition probability of movingfrom the exploration period to a preference-biased period was equallydivided between the left-selected and right-selected preference-biasedperiods. Finally, to perform inference of the participant's choice onthe test eye fixation sequence, the posterior probability of high-levelstate of the last fixation p(s_(T)|x₁, . . . , x_(T)) given the testsequence (x₁, . . . , x_(T)) was computed, which indicates theprobability of being in either the left-selected or right-selectedpreference-biased period at the end of the trial. Theleft/right-selected preference-biased period with higher probability waspredicted as the choice. This was repeated over all trials for eachparticipant to calculate the inference accuracy.

The inferences were performed in three ways. First, to examine thepercentage of fixations in a trial required for makingabove-chance-level inferences, the first 10% of the fixations were usedfor the inferences and this proportion was increased by 5% each timeuntil all fixations (100%) were used. The inference task was thereforeconducted 19 times, and the mean inference accuracy was calculated eachtime. Second, to examine how quickly a decision could be inferred,inference on increasing duration of fixation sequences was performedfrom the beginning of the trial (e.g., the fixations in the first 1second, the fixations in the first 2 seconds, etc.). Third, the gazecascade model suggested that although preferences were shaped during atrial when participants switched their fixations between the twostimuli, the fixations immediately before the end of the trial wereusually significantly biased to the preferred stimulus. Thus, thesefixations should be better predictors for participants' preference thanthe earlier fixations. Accordingly, in a separate examination only thefixations in the last 2 seconds before the decision were used to performthe inferences.

Example 12—Categorization of Individual SHMMs

One SHMM was trained for each participant and 1) a standard HMM usingthe exploration period transition matrix to represent the participant'seye movement pattern during the exploration period and 2) a standard HMMusing the preference-biased period transition matrix to represent theparticipant's eye movement pattern during the preference-biased periodwere created.

Table 2 below shows the average high-level state transition matrix.Participants started in the exploration period, and had a 55%probability to remain in the exploration period. Otherwise there was 45%probability to transition to the preference-biased period, and remainedthere until the end of the trial.

TABLE 2 High-level state transition matrix of all subjects. ExplorationPreference Prior 1.00 0.00 Exploration 0.55 0.45 Preference 0.00 1.00Next, the exploration period HMMs of the 24 participants were clusteredinto one representative HMM using the VHEM algorithm. Tables 3a and 3bbelow show the transition matrices of the representative explorationperiod HMM and preference-biased HMM, respectively.

TABLE 3a The transition matrix of the representative Exploration HMMobtained by clustering the 24 exploration period HMMs into one group.Left Right Prior 0.70 0.30 Left 0.64 0.36 Right 0.12 0.88

TABLE 3b The transition matrix of the representative preference- biasedHMM obtained by clustering the 24 preference- biased period HMMs intoone group to Chosen to Not-chosen Prior 0.53 0.47 from Chosen 0.77 0.23from Not-chosen 0.33 0.67

As indicated in Table 3a above, participants tended to first view theleft side with several fixations, and then transition to view the rightside for several fixations. After viewing the right side, theparticipants rarely looked back at the left side (12% probability). Thissuggested that participants performed a quick scan of the two sidesduring the exploration period.

Similarly, Table 3b above shows that participants in thepreference-biased period were biased to remain looking at theto-be-chosen side (77%) more often than to the not-chosen side (67%);furthermore, they were more likely to transition from thenot-chosen-side to the chosen side (33%), than vice-versa (23%).

To investigate individual differences in the gaze cascade effect,participants were clustered based on their high-level transitionmatrices into two groups. It was found that one group (group A) included11 participants and the other (group B) included 13 participants. Table4 and Tables 5a and 5b below show the high-level transition matrices,and the transition matrices of the representative exploration andpreference-biased period HMMs for the two groups.

TABLE 4 Transitions matrices of the high-level states of Group A (11participants) and Group B (13 participants). Exploration PreferenceGroup A Prior 1.00 0.00 Exploration 0.68 0.32 Preference 0.00 1.00 GroupB Prior 1.00 0.00 Exploration 0.45 0.55 Preference 0.00 1.00

TABLE 5a Transition matrices of the representative exploration period ofGroup A (11 participants) and Group B (13 participants). Left RightGroup A Prior 0.76 0.24 Left 0.67 0.33 Right 0.17 0.83 Group B Prior0.64 0.36 Left 0.60 0.40 Right 0.09 0.91

TABLE 5b Transition matrices of the representative preference-biasedperiod of Group A (11 participants) and Group B (13 participants).Chosen Not-chosen Group A Prior 0.50 0.50 Chosen 0.83 0.17 Not-chosen0.25 0.75 Group B Prior 0.54 0.46 Chosen 0.71 0.29 Not-chosen 0.39 0.61

From the high-level state transition matrix, group A had higherprobability to stay in the exploration period (68%) than group B (45%).Thus, the exploration period of group A was longer than that of group B.By examining the transition matrix during the exploration period, it wasfound that group A had a stronger tendency to start exploring from theleft side (76%) than group B (64%) and had a higher probability to staylooking at the left side (67%) than group B (60%). After switching tothe right side, group A also had higher probability to transition backto the left side (17% vs 9%).

During the preference-biased period, participants in group A showed anapparent fixation bias to stay looking at the chosen side. Morespecifically, participants in group A had a stronger tendency to keeplooking at the to-be-chosen side (83%) than group B (71%) and switchedless often between the two sides than group B.

Example 13—Cascade Plot

The analyses showed that participants were biased to look more at theside that they were about to choose during the preference-biased period.However, the clustering results showed that this difference was moreobvious for one group of participants (group A) than the other group(group B). To visualize the difference in the gaze cascade effectbetween the two participant groups, a gaze cascade plot was generated.The plot showed the probability that participants looked at the image tobe chosen during the last 2.5 seconds prior to the response. FIG. 7shows the gaze cascade plots of the two groups and their average.

As depicted in the “All” plot in FIG. 7, participants spent more time oninspecting the side that they were about to choose near the end of atrial. The proportion of time spent on the chosen side went from chancelevel 0.5 steadily up until it reached around 0.87. The probability thateach participant looked at the chosen stimuli at each time point wasestimated at a 100-millisecond (ms) interval. To test the hypothesisthat across the time intervals there was significant difference in theprobability of looking at the chosen side, and that the two groups ofparticipants differed in the time interval effect, a mixed AVOVA wasperformed on the probability of looking at the chosen side with timeinterval as a within-subject variable and group as a between-subjectvariable. The results showed a significant main effect of time interval,F(3.043, 66.938)=52.163, p<0.001, η_(p) ²=0.703, a significant maineffect of group, F(1, 22)=5.481, p=0.029, η_(p) ²=0.199, and a marginalinteraction between time interval and group, F(3.043, 66.938)=2.307,p=0.084, η_(p) ²=0.095². In addition, there was a significant lineartrend, F(1, 22)=126.657, p<0.001, η_(p) ²=0.852, and quadratic trend,F(1, 22)=19.609, p<0.001, η_(p) ²=0.471, across time intervals. Theseresults demonstrated that during the last 2.5 seconds before response,participants had a significant increase in probability of looking at thechosen side. In addition, group A had higher a probability of looking atthe chosen side than group B, indicating a stronger cascade effect. ²Greenhouse-Geisser correction was applied whenever the assumption ofsphericity was not met.

Post-hoc t-test showed that participants started to look at the chosenitem significantly above chance level at around 1100 ms before the endof the trial (i.e., the beginning of the gaze cascade effect),t(23)=2.27, p=0.033, d=0.46 (Here d refers to Cohen's d, an effect sizemeasure to indicate standardized difference between two means), untilthe end of the trial. Within this time period, the probability to lookat the chosen face rose from 58% to 87%. Furthermore, there was a shortperiod from 2200 ms and 1600 ms before the end of the trial, where theparticipants also looked at the chosen side with slightly higherprobability than chance level according to t-test (˜55%; FIG. 7).

The plots of the two participant groups showed some interestingdifferences. The participants in group A showed a stronger gaze cascadeeffect. The probability that group A looked at the chosen item reached94.5% at the end. A t-test was conducted to examine when theirprobability of viewing the chosen stimulus was above chance level. Theresult showed that this occurred at around 1000 ms before the end of atrial, t(10)=2.44, p=0.035, d=0.74, at which time point they spent about66% of their time on the chosen item. In contrast, participants in groupB had a weaker cascade effect, looking at the chosen item about 81% ofthe time at the end. A t-test indicated that the proportion of timespent on the chosen item was significantly above the chance level at 900ms before the end of a trial, t(12)=2.29, p=0.041, d=0.64, at whichpoint the probability to view the chosen side increased from 59% to 81%.Group B also exhibited a short period of slightly higher than chanceviewing (54%) of the chosen side between 2100 ms and 1600 ms before theend of the trial (FIG. 7). The proportions at each time point were alsocompared between the two groups using independent sample t-tests. It wasfound that during 700 ms before the end of a trial to the end of thetrial, group A spent significantly more time looking at the to-be-chosenitem than group B (FIG. 7). Thus, although both groups exhibited thegaze cascade effect, they differed in both magnitude and onset time,suggesting substantial individual differences in the gaze cascadeeffect. These results indicated that the EMSHMM method of the instantinvention allowed the detection of these individual differences in thegaze cascade effect through clustering participants' eye movementpatterns according to their similarities.

In addition, with the HMM based approach the similarity of eachparticipant's eye movement pattern during the preference-biased periodto the representative pattern of group A or group B could bequantitatively assessed by calculating the log likelihood of theparticipant's eye movement pattern being generated by the representativemodel. To quantify a participant's eye movement pattern along thecontinuum between the representative patterns of group A and group B,the A-B scale was defined as (LA−LB)/(|LA|+|LB|), where LA is the loglikelihood of the eye movement pattern being generated by the group Amodel, and LB is the log likelihood of the eye movement pattern beinggenerated by the group B model (Chan et al., 2018). The larger the A−Bscale, the more similar the pattern is to the representative pattern ofgroup A, and vice versa for the representative pattern of group B. Amongthe participants, a significant positive correlation was observedbetween A-B scale and gaze cascade effect as measured by the averageprobability looking at the chosen item from the onset of the effect(1000 ms prior to the response) to the end (r=0.50, p=0.012). In otherwords, the more similar participants' eye movement patterns during thepreference-biased period to the representative pattern of group A, thestronger their gaze cascade effects.

Example 14—Transition Between Exploration and Preference-Biased Periods

To examine whether the two participant groups differed in theirtransition behavior between the two cognitive states throughout a trial,the trial duration was normalized by dividing each trial into 21 timesegments and for each participant the percentage of trials, or thefrequency that the participant was in the preference-biased periodduring each time segment was examined. FIG. 8A shows the averageprobability of the two groups being in the preference-biased periodthroughout a trial. FIG. 8B shows the probability distribution of numberof fixations in the exploration period, and number of fixations in thepreference-biased period. The vertical bars indicate standard errors.

The results showed that for both groups, the probability of being in thepreference-biased period increased soon after a trial began. To test thehypothesis that group A and group B had different probabilities acrossthe time segments, a mixed ANOVA on probability of being in thepreference-biased period with time segment as the within-subjectvariable and group as the between-subject variable was conducted. Theresults showed a significant time segment effect, F(1.384,30.445)=339.091, p<0.001, η_(p) ²=0.939, a significant group effect,F(1, 22)=7.770, p=0.011, and a significant interaction between group andtime segment, F(1.384, 30.445)=10.064, p=0.001, η_(p) ²=0.314. Theseresults indicated that in general group B had higher probability to bein preference-biased period than group A, especially in the beginningtime segments of a trial (FIG. 8A). Although group A entered thepreference-biased period later, group A had a stronger gaze cascadeeffect. In contrast, group B entered the preference-biased periodearlier, but had a weaker gaze cascade effect.

A t-test was used to test the hypothesis that group A and group Bdiffered in average number of fixations per trial. It was found thatgroup A (M=29.0) had a significantly larger average number of fixationsin a trial than group B (M=13.3), t(22)=4.44, p<0.001, d=1.75. Group Aalso had a larger average number of fixations in the exploration period(group A: M=3.38; group B: M=1.88), t(22)=8.46, p<0.001, d=3.37, and inthe preference-biased state (group A: M=25.61; group B: M=11.5),t(22)=4.10, p<0.001, d=1.62 (FIG. 8B). However, after normalizing forthe total number of fixations, group A (M=0.147) and group B (M=0.173)did not differ in the fraction of fixations in the exploration orpreference-biased period, t(22)=−1.11; p=0.28; d=0.45.

Further, the hypothesis was tested that group A and group B differed inthe number of eye gaze switches between stimuli during a trial. Group A(M=5.66) had more switches than group B (M=4.10) on average, t(22)=2.81,p=0.01, d=1.12. However, this effect was mainly due to group A havingmore fixations in a trial in general. After normalizing for the totalnumber of fixations in a trial, group A (M=0.211) had a smaller fractionof eye fixations that involved switching between stimuli than group B(M=0.328), t(22)=−7.20, p<0.001, d=2.97. This effect suggested thatgroup A tended to explore a stimulus longer before switching to theother stimulus than group B. Note that since the clustering was onlybased on participants' cognitive (high-level) state transitions, thesedifferences in number of fixations per trial and frequency of switchbetween stimuli emerged naturally as a result of the clustering.

Example 15—Inference of Participants' Preference Choices

It was also explored whether the models could be used to infer anindividual's preference choice in each trial given partial eye movementdata. The fixations from the first 10% of the trial according tonormalized duration ere used and were gradually increased at a step of5%. FIG. 9A shows the average inference accuracies of the two groupsusing partial fixations in a trial, selected as a percentage of eachtrial's duration from the beginning. FIG. 9B shows the average inferenceaccuracies of the two groups using different window lengths startingfrom the beginning of the trial (note that participants had differenttrial lengths, with the average length 5.65±2.57 seconds; Group A hadaverage trial length of 7.67±2.32 seconds, while Group B had averagetrial length of 3.93±1.13 seconds). The red, blue, and green stars onthe top indicated the data points at which the accuracy wassignificantly higher than the chance level based on t-test, for eachgroup and all participants respectively.

As shown in FIG. 9A, when using the first 75% to 90% of the fixations ina trial, the average inference accuracy was higher for group B than forgroup A. In contrast, when the first 95% or all fixations of a trialwere used, the inference accuracy for group A was higher than that forgroup B. To test the hypothesis that group B had higher inferenceaccuracy when 75% to 90% of the fixations were used whereas group A hadhigher inference accuracy when 95% to 100% of fixations were used, amixed ANOVA on inference accuracy was conducted with amount of fixations(75% to 100% at a step of 5%) as the within-subject variable and groupas the between-subject variable. The results showed a main effect ofamount of fixations, F(1.356, 29.840)=30.778, p<0.001, η_(p) ²=0.583,and an interaction between amount of fixations and group, F(1.356,29.840)=5.657, p=0.016, η_(p) ²=0.205. The main effect of group was notsignificant, F(1, 22)=0.077, n.s. These results indicated that, as shownin FIG. 9A, group B had higher inference accuracy when a smaller amountof fixations were used (75% to 90%), whereas group A had higherinference accuracy when a larger amount of fixations were used. When theinference accuracies were compared against the chance-level (0.5) usingt-test, it was found that when using the first 75% to 85% of thefixations, group B's inference accuracy was significantly above thechance-level, whereas group A's inference accuracy was not. When usingthe first 90% to 100% of the fixations, inference accuracy wassignificantly above the chance-level for both groups (FIG. 9A). In otherwords, participants in group B revealed their biases to the preferredstimulus earlier than group A.

To examine the actual time when a group's accuracy became significantlyabove chance, FIG. 9B plots the average inference accuracy using timewindows (in seconds) starting from the beginning of the trial. After thestart of the trial, Group B's inference accuracy increased more rapidlythan Group A, and became significantly above chance at around 3 seconds,and then saturated at around 4 seconds. In contrast, Group A's inferenceaccuracy increased slowly, became above chance-level at around 6seconds, and saturated at around 10 seconds. Although Group A's accuracyincreased more slowly, it saturated at a higher value than Group B.

Example 16—Last 2 Seconds Before Response

The gaze cascade effect suggested that eye movement patterns immediatelybefore a preference decision response may provide a strong cue forinferring the preference. According to FIG. 7, participants started toshow a tendency of looking at the chosen side more often at around 2seconds before response. Thus, the accuracy of inferring participants'preferences was examined using the fixations during the last 2 secondsbefore the response. FIG. 10A shows the average inference accuracies ofthe two groups using fixations in the last 2 seconds until the response.FIG. 10B shows the more similar participants' eye movement patterns tothe representative pattern of group A (i.e. the further to the right ofthe X-axis), the higher the inference accuracy.

First, the hypothesis was tested that for participants in both group Aand group B, the average accuracy of inferring their preferencedecisions using the final 2 seconds was significantly above the chancelevel, and results supported this hypothesis (FIG. 10A): group A,M=0.93, t(10)=15.89, p<0.001, d=4.79; group B, M=0.71, t(12)=3.79,p=0.003, d=1.05. In addition, the hypothesis was tested that theinference accuracy of group A was significantly higher than that ofgroup B. The result supported the hypothesis, t(22)=3.26, p=0.004,d=1.33. In addition, the more similar participants' eye movementpatterns during the preference biased period to the representativepattern of group A, as opposed to that of group B (as measured in A-Bscale), the higher the inference accuracy (r=0.47, p=0.02; FIG. 10B).This result was consistent with the observation that group A exhibited astronger gaze cascade effect (FIG. 7). The clustering of the two groupswas completely based on the eye movement data alone, and thus the groupdifference in inference accuracy emerged naturally as the result of theclustering.

It was also tested whether a regular HMM without inferring participants'cognitive state transition, such as that used in the previous EMHMMapproach (Chuk et al., 2014), would be able to reveal participants'preference choices. To this end, the same inference task was performedusing regular HMMs in the EMHMM approach. As shown in FIG. 11, theaverage inference accuracy of SHMMs (M=0.81) was higher than HMMs(M=0.64) for all subjects, t(46)=2.71, p=0.009, d=0.78. A two-way ANOVAon inference accuracy was conducted with group and model (SHMM/HMM) asthe independent variables. The results showed a significant main effectof group, F(1, 44)=5.04, p=0.03, η_(p) ²=0.096 and a significant maineffect of model, F(1, 44)=8.75, p=0.005, η_(p) ²=0.166. There was nointeraction between group and model, F(1, 44)=1.91, p=0.17. This resultdemonstrated again the advantage of EMSHMM for modeling eye movementpatterns in tasks that involve cognitive state changes. FIG. 11 showsthe average inference accuracy for all participants using SHMMs andregular HMMs when using the last 2 seconds of the trials.

Example 17—Relevance of the Study Results

The instant invention presents a novel method, EMSHMM, for modeling eyemovement patterns in tasks that involve cognitive state changes. Similarto the previous hidden Markov modeling approach to eye movement dataanalysis EMHMM, the EMSHMM approach has several advantages overtraditional eye movement data analysis methods such as ROI or fixationheat map analysis, including the ability to account for individualdifferences in both spatial and temporal dimensions of eye movements(i.e., through discovering personalized ROIs and transitionprobabilities among the ROIs) and to quantitatively assess thesedifferences. In contrast to EMHMM, which uses a single regular HMM tomodel eye movements and assumes a participant's strategy is consistentthroughout a trial, the EMSHMM approach uses multiple low-level HMMscorresponding to different strategies/cognitive states, and ahigher-level state sequence to capture the transitions among differentstrategies/cognitive states. Thus, it is especially suitable foranalyzing eye movement data in complex tasks that involve cognitivestate changes such as decision-making tasks.

To demonstrate the advantages of using the EMSHMM approach, a preferencedecision-making task was conducted, in which participants viewed twofaces with similar attractiveness ratings and decided which one theypreferred. Previously two different eye movement patterns at differentstates of a trial had been observed; they usually began with exploringboth alternatives and then focused on the one preferred by the end ofthe trial and these two eye movement patterns were associated withdifferent cognitive states. In the instant EMSHMM approach, it wasassumed that the two eye movement patterns were associated with twocognitive states, exploration period and preference-biased period,respectively. A switching HMM (SHMM) was used to summarize aparticipant's eye movement pattern in the preference decision-makingtask. The SHMM contained two ROIs that corresponded to the two faces ofchoice; two low-level HMMs that summarized the eye movement patternsduring the exploration and preference-biased periods, respectively; anda high-level state sequence that captured the transitions between thetwo cognitive states.

A summary of all participants' high-level/cognitive state transitionsshowed that on average they had a 55% probability to remain in theexploration period, and 45% probability to transition to thepreference-biased period, and remained there until the end of the trial(Table 2). When all participants' exploration period HMMs weresummarized in one representative model, it was found that participantshad a bias to start from looking at the stimulus on the left side andremain exploring there, and then switch to the right side (Table 3a). Incontrast, when all participants' preference-biased period HMMs weresummarized in one representative model, it was found that participantslooked more often at the to-be-chosen, preferred stimulus (Table 3b).When the percentage of time that participants were looking at theto-be-chosen stimulus before the end of a trial was plotted, it showed asteady increase at about 1.5 seconds before the end, demonstrating agaze cascade effect (FIG. 7).

When participants' SHMMs were clustered into two groups according totheir cognitive state transitions, one group (group A) showed a strongerand earlier gaze cascade effect than the other group (group B; FIG. 7).The two groups also showed interesting differences in the temporaldynamics of eye movement patterns throughout a trial. More specifically,participants in group A entered the preference-biased period later thangroup B (FIG. 8A) but had a stronger cascade effect. In addition, groupA's preference over the two alternatives could not be inferred withabove-chance performance using early fixations of a trial until thefirst 90% of the fixations were used. In contrast, group B's preferencecould be inferred with above-chance performance with only the first 75%of the fixations (FIG. 9A). However, when only the fixations during thefinal 2 seconds before the decision response were used, group A'spreference was inferred with a higher accuracy than group B. Thisphenomenon showed that although participants in group A revealed theirpreference in the eye movement patterns later in a trial than group B,their eye movement patterns contained more information for inferringtheir preferences. Recent research has suggested that indecisiveness, ordecisional procrastination, is associated with informational tunnelvision: indecisive individuals tend to gather more information about theitem that is eventually chosen while ignoring information about otheralternatives. Accordingly, in the instant study participants in group Bhave exhibited a more ‘indecisive’ eye movement pattern than group A,since they entered the preference-biased period earlier, spentproportionally less time in the exploration period, and switched morefrequently between the two stimuli for choice, which may becharacteristics of informational tunnel vision. Thus, there exists arelationship between eye movement pattern similarity (as assessed usingthe EMHMM/EMSHMM approach) to the representative pattern of group B andparticipants' personality measures related to indecisiveness.

These individual differences in eye movement pattern and cognitive styleduring decision making have not been reported before in the literature.More specifically, previous studies only observed that decisions wererelated to the final fixations in a trial as revealed in the gazecascade effect. However, using the instant method it can be shown thatparticipants' preference can be inferred significantly earlier than theexhibition of the gaze cascade effect, and for some participants (e.g.,group B) this inference could achieve above-chance level performancewith only the first 75% of the fixations. Interestingly, theseparticipants also tended to show a weaker gaze cascade effect. Thesefindings demonstrate the importance of taking individual differencesinto account in the understanding of human decision-making behavior.Importantly, while regular HMMs without cognitive state transitions inan EMHMM approach could also account for individual differences in eyemovement patterns, the accuracy in inferring participants' preferencechoices using EMSHMM is surprisingly and significantly higher than thatusing EMHMM. Advantageously, EMSHMM can better capture the cognitiveprocesses involved in the task and consequently lead to higher inferenceaccuracy.

Further advantageously, the instant methods can deduce participants'preference during a decision making process from eye gaze transitioninformation alone. In contrast, previous methods needed to combine eyemovement measures with other information such as additionalphysiological measures or attended visual features including integratedskin conductance, blood volume pulse, pupillary response.

Furthermore, while previous methods reached an average accuracy of 81%,the instant methods deduce participants' preference from eye gazetransition information alone using EMSHMM with more than 90% accuracies.

In addition, EMSHMM provides quantitative measures of similarities amongindividual eye movement patterns by calculating the log-likelihood ofone's eye movement data being generated by a representative HMM. Forexample, it was shown that the similarity of participants' eye movementpatterns during the preference-biased period to the representativepattern of group A as opposed to that of group B (as measured in A-Bscale) was positively correlated with the gaze cascade effect andinference accuracy using fixations during the final 2 seconds beforeresponse. In addition to examining the relationship between eye movementpatterns and other psychological measures, using methods of the instantinvention it can also be determined how the eye movement patternsimilarity measure is modulated by factors related to decision makingstyles, such as gender, cultural, sleep loss, etc. Using EMHMM, it wasshown that eye movement pattern similarity to an eye-centered, analyticpattern during face recognition is associated with better recognitionperformance whereas similarity to a nose-centered, holistic patterns iscorrelated with cognitive decline in older adults (see e.g., Chuk, Chan,et al., 2017; Chuk, Crookes, et al., 2017; Chan et al., 2018), thatindividuals with insomnia symptoms exhibit eye movement patterns moresimilar to a representative nose-mouth pattern during facial expressionjudgments as compared with healthy controls (Zhang, Chan, Lau, & Hsiao,2019). Advantageously, the EMSHMM of the instant invention can be usedto examine how eye movement patterns are associated with otherpsychological measures and factors that may affect eye movement patternsin more complex tasks that involve cognitive state changes.

While the analysis using methods of the instant invention so far focusedon the eye gaze transition behavior between two stimuli of choice in thepreference decision-making task by using only two ROIs, with eachcorresponding to a stimulus, further methods are provided to analysiseye movement pattern, e.g., ROIs (low-level states) and transitionprobabilities among the ROIs, on each stimulus to capture individualdifferences in information extraction in addition to gaze transition indecision-making behavior. Previous studies showed that participants havepreferred fixated features or fixation locations during subjectivedecision-making. For example, it was found that attractive andunattractive features received more attention than those withintermediate attractiveness and brands located at the center of a shelfin shops were more likely to be chosen. Because individuals differ inhow they obtain information from the stimuli of choice during decisionmaking, or in a cognitive task in general, the methods of the instantinvention using an EMSHMM toolbox (see Chuk et al., 2014) capture theseindividual differences by inferring personalized ROIs on each stimulususing a Gaussian mixture model approach and enable the determination ofthe optimal number of ROIs for each participant trough the Bayesianmethod. Advantageously, using a larger number of high-level states, theEMSHMM method is able to discover more fine-grained cognitive states inbetween the exploration and preference-biased periods in terms ofsimilarity. For example, in more complex cognitive tasks such as drivingor cooking, with a large number of high-level states the instant methodenables the detection of more discrete cognitive states essential to thetask and their associated eye movement patterns.

The SHMM can represent differences in transition matrices within a trial(intra-trial differences), while other methods, e.g., the mixed HMM byAltman (2007), add random effects to the HMM. In particular, randomeffects are added to the emission density means and to thelog-probabilities of the transition matrix and prior. This allowsinter-subject or inter-trial differences to be represented in a singlemodel. Thus, in some embodiments of the invention the SHMM are extendedto add random effects to model inter-subject differences in a singlemodel.

In summary, in some embodiments, the novel EMSHMM-based methods of theinstant invention analyze eye movement data in tasks that involvecognitive state changes where for each participant an SHMM is used tocapture between cognitive state transitions during the task, with eyemovement patterns during each cognitive state being summarized using anHMM.

In some embodiments, the EMSHMM of the invention is applied to a facepreference decision-making task. In specific embodiments, the EMSHMM ofthe invention identifies two common eye movement patterns from theparticipants, where one pattern entered the preference-biased cognitivestate later, showed a stronger gaze cascade effect immediately beforethe decision response, and allowed the determination of the preferencedecision later in a trial and the other pattern revealed the preferencedecision much earlier in a trial, spent more time in thepreference-biased cognitive state, had a weaker gaze cascade effect inthe end, and led to a lower decision response inference accuracy.

These surprising differences emerged naturally as the result ofclustering based on eye movement data alone, and were not revealed byany existing methods in the literature. As compared with previousapproaches, the EMSHMM method of the invention is unexpectedly superiorat capturing eye movement behavior in the task and infers participants'decision responses with higher accuracy. In addition, EMSHMM providesquantitative measures of similarities among individual eye movementpatterns, and thus is particularly suitable for studies using eyemovements to examine individual differences in cognitive processes,making a significant impact on the use of eye tracking to studycognitive behavior across disciplines.

It should be understood that the examples and embodiments describedherein are for illustrative purposes only and that various modificationsor changes in light thereof will be suggested to persons skilled in theart and are to be included within the spirit and purview of thisapplication and the scope of the appended claims. In addition, anyelements or limitations of any invention or embodiment thereof disclosedherein can be combined with any and/or all other elements or limitations(individually or in any combination) or any other invention orembodiment thereof disclosed herein, and all such combinations arecontemplated with the scope of the invention without limitation thereto.

REFERENCES

-   Altman, R. (2007). Mixed hidden Markov models: An extension of the    hidden Markov model to the longitudinal data setting. Journal of the    American Statistical Association, 102, 201-210.-   Andrews, T. J., & Coppola, D. M. (1999). Idiosyncratic    characteristics of saccadic eye movements when viewing different    visual environments. Vision research, 39(17), 2947-2953.-   Castelhano, M. S., & Henderson, J. M. (2008). Stable individual    differences across images in human saccadic eye movements. Canadian    Journal of Experimental Psychology, 62(1), 1-14.-   Chan, C. Y. H., Chan, A. B., Lee, T. M. C., & Hsiao, J. H. (2018).    Eye movement patterns in face recognition are associated with    cognitive decline in older adults. Psychonomic Bulletin & Review,    25(6), 2200-2207.-   Chuk, T., Chan, A. B., & Hsiao, J. H. (2014). Understanding eye    movements in face recognition using hidden Markov models. Journal of    Vision, 14(11):8, 1-14.-   Chuk, T., Chan, A. B., & Hsiao, J. H. (2017). Is having similar eye    movement patterns during face learning and recognition beneficial    for recognition performance? Evidence from hidden Markov modeling.    Vision Research, 141, 204-216.-   Chuk, T., Crookes, K., Hayward, W. G., Chan, A. B., & Hsiao, J. H.    (2017). Hidden Markov model analysis reveals the advantage of    analytic eye movement patterns in face recognition across cultures.    Cognition, 169, 102-117.-   Coviello, E., Chan, A. B., & Lanckriet, G. R. (2014). Clustering    hidden Markov models with variational HEM. The Journal of Machine    Learning Research, 15(1), 697-747.-   Govaert, G., & Nadif, M. (2013). Co-clustering: models, algorithms    and applications. ISTE, Wiley. ISBN 978-1-84821-473-6.-   Kanan, C., Bseiso, D. N., Ray, N. A., Hsiao, J. H., &    Cottrell, G. W. (2015). Humans have idiosyncratic and task-specific    scanpaths for judging faces. Vision research, 108, 67-76.-   Lau, E. Y. Y., Eskes G. A., Morrison, D. L., Rajda, M., Spun, K. F.    (2010). Executive function in patients with obstructive sleep apnea    treated with continuous positive airway pressure. J. Int.    Neuropsych. Soc., 16, 1077-1088.-   MacQueen, J. (1967). Some methods for classification and analysis of    multivariate observations. In Proceedings of the Fifth Berkeley    Symposium on Mathematical Statistics and Probability (pp. 281-297′.    Berkeley, Calif.: University of California Press.-   Phillips, L. H., Wynn, McPherson, S., & Gilhooly, K. J. (2001).    Mental planning and the Tower of London task. Q. J. Exp. Psychol.—A,    54, 579-597-   Poynter, W., Barber, M., Inman, J., & Wiggins, C. (2013).    Individuals exhibit idiosyncratic eye-movement behavior profiles    across tasks. Vision research, 89, 32-38.-   Ridderinkhof, K. R., Band, G. P., & Logan, D. (1999). A study of    adaptive behavior: effects of age and irrelevant information on the    ability to inhibit one's actions. Acta Psychol., 101, 315-337.-   Shimojo, S., Simion, C., Shimojo, E., & Scheier, C. (2003). Gaze    bias both reflects and influences preference. Nature neuroscience,    6(12), 1317-1322.-   Wu, D. W. L., Bischof, W. F., Anderson, N. C., Jakobsen, T., &    Kingstone, A. (2014). The influence of personality on social    attention. Personality and Individual Differences, 60, 25-29.-   Yeung, P. Y., Wong, L. L., Chan, C. C., Leung, J. L., & Yung, C. Y.    (2014). A validation study of the Hong Kong version of Montreal    Cognitive Assessment (HK-MoCA) in Chinese older adults in Hong Kong.    Hong Kong Med. J., 20(6), 504-510.-   Zhang, J., Chan, A. B., Lau, E. Y. Y., & Hsiao, J. H. (2019).    Individuals with insomnia misrecognize angry faces as fearful faces    while missing the eyes: An eye-tracking study. Sleep, 42(2), zsy220.

1-7. (canceled)
 8. An eye movement analysis with hidden Markov model(EMHMM) with co-clustering comprising: a vector of prior values ofhidden states, a transition matrix of the hidden states, and a Gaussianemission for each hidden state; wherein the prior values indicate theprobabilities of time-series data starting from a corresponding hiddenstate; the transition matrix indicates the transition probabilitiesbetween any two hidden states; and the Gaussian emissions indicate theprobabilistic associations between an observed time-series data and ahidden state; and a co-clustering data mining technique; wherein theEMHMM with co-clustering analyzes eye movement data involving stimuliwith different feature layouts.
 9. An eye movement analysis withswitching hidden Markov model (EMSHMM) comprising: a vector of priorvalues of hidden states, a transition matrix of the hidden states, and aGaussian emission for each hidden state; wherein the prior valuesindicate the probabilities of time-series data starting from acorresponding hidden state; the transition matrix indicates thetransition probabilities between any two hidden states; and the Gaussianemissions indicate the probabilistic associations between an observedtime-series data and a hidden state; and two levels of hidden statesequences comprising: several low-level hidden state sequence modelsthat are used to learn the eye movement pattern of several cognitivestates and a high-level hidden state sequence model that acts like aswitch that captures the transitions between cognitive states; whereinthe EMSHMM analyzes eye movement data in complex tasks that involvecognitive state changes, wherein the EMSHMM is based on aggregated SHMMfrom the two separately trained SHMMs.
 10. A method for determining acognitive style and/or cognitive ability in a subject, the methodcomprising: collecting eye movement data of a subject; computing regionsof interest (ROIs) based on the eye movement data; measuring transitiontimes between the ROIs; calculating transition probabilities among theROIs; clustering eye movements into groups using co-clustering;assessing each individual's data log-likelihoods using the co-clusteringeye movement group models; measuring executive function using TOL,visual attention using Flanker Task, working memory using Verbal andVisuospatial Two-Back Task; determining the relationship between datalog-likelihood measures and measured executive function, visualattention, and working memory; and determining a cognitive style and/orcognitive ability of the subject based on the data log-likelihoodmeasures using the co-clustering eye movement group models, or the eyemovement group to which the subject's eye movement was co-clustered. 11.A method for determining the cognitive state in a subject, the methodcomprising: collecting eye movement data of a subject; computing regionsof interest (ROIs) based on the eye movement data; measuring transitiontimes between the ROIs; calculating transition probabilities using theEMSHMM according to claim 9; and determining the cognitive state of thesubject based on the EMSHMM results.
 12. A method for inferring asubject's choice, the method comprising: providing two image choices;collecting eye movement data of a subject looking at the two imagechoices; training one switching hidden Markov model (SHMM) for a firstchosen image and one SHMM for a second chosen image; generating anaggregated SHMM from the two trained SHMMs; calculating the probabilityof a last fixation based on the aggregated SHMM; and inferring a choicebased on the calculated probability and the subject's last fixation. 13.An electronic device for determining a cognitive style in a subject, thedevice comprising: a camera configured to capture a facial image; and aprocessor configured to detect a center position of an eye within afacial image via the camera, to determine an eye gaze position based onthe center position and a pupil position, to analyze the eye gazeposition in consecutively captured facial images, and to measure an eyemovement patterns based on the eye gaze positions of the consecutivelycaptured facial images; wherein the processor is further configured tocalculate a log likelihood of an eye movement pattern and to assign acertain cognitive style to the measured eye movement pattern if itmatches the eye movement pattern of a representative group of subjectshaving the certain cognitive style.
 14. The device according to claim13, wherein the processor is further configured to receive externaluser-entered data that assign certain eye movement patterns with certainexternal task criteria and/or cognitive state criteria; and theprocessor is configured to assign a cognitive state to the subject basedon the external task criteria and/or cognitive state criteria and thesubject's eye movement patterns.